Creating new pipeline using seurat v4.0.2 available 2021.06.08

Load libraries required for Seuratv4

knitr::opts_knit$set(root.dir = "~/Desktop/10XGenomicsData/msAggr_scRNASeq/")
library(dplyr)

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union
library(Seurat)
Registered S3 methods overwritten by 'htmltools':
  method               from         
  print.html           tools:rstudio
  print.shiny.tag      tools:rstudio
  print.shiny.tag.list tools:rstudio
Registered S3 method overwritten by 'data.table':
  method           from
  print.data.table     
Registered S3 method overwritten by 'htmlwidgets':
  method           from         
  print.htmlwidget tools:rstudio
Attaching SeuratObject
library(patchwork)
library(ggplot2)
library(clustree)
Loading required package: ggraph

store session info

sink("msAggr_sct-v1.20210615")
sessionInfo()
sink()

A note about using SCTransform versus ScaleData

https://bioconductor.org/packages/3.10/workflows/vignettes/simpleSingleCell/inst/doc/batch.html#62_for_gene-based_analyses >You can also normalize and scale data for the RNA assay. There are numerous resources on this, but Aaron Lun describes why the original log-normalized values should be used for DE and visualizations of expression quite well here: > >For gene-based procedures like differential expression (DE) analyses or gene network construction, it is desirable to use the original log-expression values or counts. The corrected values are only used to obtain cell-level results such as clusters or trajectories. Batch effects are handled explicitly using blocking terms or via a meta-analysis across batches. We do not use the corrected values directly in gene-based analyses, for various reasons: > >It is usually inappropriate to perform DE analyses on batch-corrected values, due to the failure to model the uncertainty of the correction. This usually results in loss of type I error control, i.e., more false positives than expected. > >The correction does not preserve the mean-variance relationship. Applications of common DE methods like edgeR or limma are unlikely to be valid. > >Batch correction may (correctly) remove biological differences between batches in the course of mapping all cells onto a common coordinate system. Returning to the uncorrected expression values provides an opportunity for detecting such differences if they are of interest. Conversely, if the batch correction made a mistake, the use of the uncorrected expression values provides an important sanity check. > >In addition, the normalized values in SCT and integrated assays don’t necessary correspond to per-gene expression values anyway, rather containing residuals (in the case of the scale.data slot for each).

Load 4 populations into single seurat object

SET SEED?????!!!!!

Set global variables

projectName <- "msAggr_sct"
jackstraw.dim <- 40
source("msAggr_AnalysisCode/read_10XGenomics_data.R")
setwd("../cellRanger/") # temporarily changing wd only works if you run the entire chunk at once
Warning: The working directory was changed to /Users/heustonef/Desktop/10XGenomicsData/cellRanger inside a notebook chunk. The working directory will be reset when the chunk is finished running. Use the knitr root.dir option in the setup chunk to change the working directory for notebook chunks.
data_file.list <- read_10XGenomics_data(sample.list = c("LSKm2", "CMPm2", "MEPm", "GMPm"))
seurat.object.data<-Read10X(data_file.list)
seurat.object<- CreateSeuratObject(counts = seurat.object.data, min.cells = 3, min.genes = 200, project = projectName)

Clean up to free memory

remove(seurat.object.data)

Add mitochondrial metadata and plot some basic features

seurat.object[["percent.mt"]] <- PercentageFeatureSet(seurat.object, pattern = "^mt-")
VlnPlot(seurat.object, features = c("nFeature_RNA", "nCount_RNA", "percent.mt"), ncol = 3, pt.size = 0, fill.by = 'orig.ident', )

plot1 <- FeatureScatter(seurat.object, feature1 = "nCount_RNA", feature2 = "percent.mt", group.by = "orig.ident", pt.size = 0.01)
plot2 <- FeatureScatter(seurat.object, feature1 = "nCount_RNA", feature2 = "nFeature_RNA", group.by = "orig.ident", pt.size = 0.01)
plot1 + plot2

remove low quality cells ** require: nFeature_RNA between 200 and 4000 (inclusive) –require: percent.mt <5–???

** In previous run it’s possible that we’ll need to limit percent.mt because of MEPm weirdness. Will have to see if the additional 2000 MEPm cells found by cellranger v6

print(paste("original object:", nrow(seurat.object@meta.data), "cells", sep = " "))
[1] "original object: 40815 cells"
seurat.object <- subset(seurat.object, 
                                                subset = nFeature_RNA >=200 & 
                                                    nFeature_RNA <= 4000
                                                )
print(paste("new object:", nrow(seurat.object@meta.data), "cells", sep = " "))
[1] "new object: 40815 cells"
seurat.object <- PercentageFeatureSet(seurat.object, pattern = '^mt-', col.name = "percent.mt")

SCTransform

As previously mentioned, SCTransform() is best for batch correction, whereas NormalizeData(), ScaleData(), and FindVariableFeatures() are best for DGE. I’m struggling to make stable clusters, so I’m jumping ahead to SCTransform().
It does appear that SCTransform() data are stored in the SCT assay slot, which does not overwrite other normalization slots. It will become the default slot once populated.

seurat.object <- SCTransform(seurat.object, 
                                                         method = "glmGamPoi", 
                                                         vars.to.regress = c("nCount_RNA", "nFeature_RNA")
                                                         ) # method = "glmGamPoi" speeds process
Calculating cell attributes from input UMI matrix: log_umi
Variance stabilizing transformation of count matrix of size 17709 by 40815
Model formula is y ~ log_umi
Get Negative Binomial regression parameters per gene
Using 2000 genes, 5000 cells

  |                                                                                                                         
  |                                                                                                                   |   0%
  |                                                                                                                         
  |=============================                                                                                      |  25%
  |                                                                                                                         
  |==========================================================                                                         |  50%
  |                                                                                                                         
  |======================================================================================                             |  75%
  |                                                                                                                         
  |===================================================================================================================| 100%
Found 108 outliers - those will be ignored in fitting/regularization step

Second step: Get residuals using fitted parameters for 17709 genes

  |                                                                                                                         
  |                                                                                                                   |   0%
  |                                                                                                                         
  |===                                                                                                                |   3%
  |                                                                                                                         
  |======                                                                                                             |   6%
  |                                                                                                                         
  |==========                                                                                                         |   8%
  |                                                                                                                         
  |=============                                                                                                      |  11%
  |                                                                                                                         
  |================                                                                                                   |  14%
  |                                                                                                                         
  |===================                                                                                                |  17%
  |                                                                                                                         
  |======================                                                                                             |  19%
  |                                                                                                                         
  |==========================                                                                                         |  22%
  |                                                                                                                         
  |=============================                                                                                      |  25%
  |                                                                                                                         
  |================================                                                                                   |  28%
  |                                                                                                                         
  |===================================                                                                                |  31%
  |                                                                                                                         
  |======================================                                                                             |  33%
  |                                                                                                                         
  |==========================================                                                                         |  36%
  |                                                                                                                         
  |=============================================                                                                      |  39%
  |                                                                                                                         
  |================================================                                                                   |  42%
  |                                                                                                                         
  |===================================================                                                                |  44%
  |                                                                                                                         
  |======================================================                                                             |  47%
  |                                                                                                                         
  |==========================================================                                                         |  50%
  |                                                                                                                         
  |=============================================================                                                      |  53%
  |                                                                                                                         
  |================================================================                                                   |  56%
  |                                                                                                                         
  |===================================================================                                                |  58%
  |                                                                                                                         
  |======================================================================                                             |  61%
  |                                                                                                                         
  |=========================================================================                                          |  64%
  |                                                                                                                         
  |=============================================================================                                      |  67%
  |                                                                                                                         
  |================================================================================                                   |  69%
  |                                                                                                                         
  |===================================================================================                                |  72%
  |                                                                                                                         
  |======================================================================================                             |  75%
  |                                                                                                                         
  |=========================================================================================                          |  78%
  |                                                                                                                         
  |=============================================================================================                      |  81%
  |                                                                                                                         
  |================================================================================================                   |  83%
  |                                                                                                                         
  |===================================================================================================                |  86%
  |                                                                                                                         
  |======================================================================================================             |  89%
  |                                                                                                                         
  |=========================================================================================================          |  92%
  |                                                                                                                         
  |=============================================================================================================      |  94%
  |                                                                                                                         
  |================================================================================================================   |  97%
  |                                                                                                                         
  |===================================================================================================================| 100%
Computing corrected count matrix for 17709 genes

  |                                                                                                                         
  |                                                                                                                   |   0%
  |                                                                                                                         
  |===                                                                                                                |   3%
  |                                                                                                                         
  |======                                                                                                             |   6%
  |                                                                                                                         
  |==========                                                                                                         |   8%
  |                                                                                                                         
  |=============                                                                                                      |  11%
  |                                                                                                                         
  |================                                                                                                   |  14%
  |                                                                                                                         
  |===================                                                                                                |  17%
  |                                                                                                                         
  |======================                                                                                             |  19%
  |                                                                                                                         
  |==========================                                                                                         |  22%
  |                                                                                                                         
  |=============================                                                                                      |  25%
  |                                                                                                                         
  |================================                                                                                   |  28%
  |                                                                                                                         
  |===================================                                                                                |  31%
  |                                                                                                                         
  |======================================                                                                             |  33%
  |                                                                                                                         
  |==========================================                                                                         |  36%
  |                                                                                                                         
  |=============================================                                                                      |  39%
  |                                                                                                                         
  |================================================                                                                   |  42%
  |                                                                                                                         
  |===================================================                                                                |  44%
  |                                                                                                                         
  |======================================================                                                             |  47%
  |                                                                                                                         
  |==========================================================                                                         |  50%
  |                                                                                                                         
  |=============================================================                                                      |  53%
  |                                                                                                                         
  |================================================================                                                   |  56%
  |                                                                                                                         
  |===================================================================                                                |  58%
  |                                                                                                                         
  |======================================================================                                             |  61%
  |                                                                                                                         
  |=========================================================================                                          |  64%
  |                                                                                                                         
  |=============================================================================                                      |  67%
  |                                                                                                                         
  |================================================================================                                   |  69%
  |                                                                                                                         
  |===================================================================================                                |  72%
  |                                                                                                                         
  |======================================================================================                             |  75%
  |                                                                                                                         
  |=========================================================================================                          |  78%
  |                                                                                                                         
  |=============================================================================================                      |  81%
  |                                                                                                                         
  |================================================================================================                   |  83%
  |                                                                                                                         
  |===================================================================================================                |  86%
  |                                                                                                                         
  |======================================================================================================             |  89%
  |                                                                                                                         
  |=========================================================================================================          |  92%
  |                                                                                                                         
  |=============================================================================================================      |  94%
  |                                                                                                                         
  |================================================================================================================   |  97%
  |                                                                                                                         
  |===================================================================================================================| 100%
Calculating gene attributes
Wall clock passed: Time difference of 2.660115 mins
Determine variable features
Place corrected count matrix in counts slot
Regressing out nCount_RNA, nFeature_RNA

  |                                                                                                                         
  |                                                                                                                   |   0%
  |                                                                                                                         
  |=                                                                                                                  |   0%
  |                                                                                                                         
  |=                                                                                                                  |   1%
  |                                                                                                                         
  |==                                                                                                                 |   1%
  |                                                                                                                         
  |==                                                                                                                 |   2%
  |                                                                                                                         
  |===                                                                                                                |   2%
  |                                                                                                                         
  |===                                                                                                                |   3%
  |                                                                                                                         
  |====                                                                                                               |   3%
  |                                                                                                                         
  |====                                                                                                               |   4%
  |                                                                                                                         
  |=====                                                                                                              |   4%
  |                                                                                                                         
  |=====                                                                                                              |   5%
  |                                                                                                                         
  |======                                                                                                             |   5%
  |                                                                                                                         
  |======                                                                                                             |   6%
  |                                                                                                                         
  |=======                                                                                                            |   6%
  |                                                                                                                         
  |========                                                                                                           |   7%
  |                                                                                                                         
  |=========                                                                                                          |   7%
  |                                                                                                                         
  |=========                                                                                                          |   8%
  |                                                                                                                         
  |==========                                                                                                         |   8%
  |                                                                                                                         
  |==========                                                                                                         |   9%
  |                                                                                                                         
  |===========                                                                                                        |   9%
  |                                                                                                                         
  |===========                                                                                                        |  10%
  |                                                                                                                         
  |============                                                                                                       |  10%
  |                                                                                                                         
  |============                                                                                                       |  11%
  |                                                                                                                         
  |=============                                                                                                      |  11%
  |                                                                                                                         
  |=============                                                                                                      |  12%
  |                                                                                                                         
  |==============                                                                                                     |  12%
  |                                                                                                                         
  |==============                                                                                                     |  13%
  |                                                                                                                         
  |===============                                                                                                    |  13%
  |                                                                                                                         
  |================                                                                                                   |  14%
  |                                                                                                                         
  |=================                                                                                                  |  14%
  |                                                                                                                         
  |=================                                                                                                  |  15%
  |                                                                                                                         
  |==================                                                                                                 |  15%
  |                                                                                                                         
  |==================                                                                                                 |  16%
  |                                                                                                                         
  |===================                                                                                                |  16%
  |                                                                                                                         
  |===================                                                                                                |  17%
  |                                                                                                                         
  |====================                                                                                               |  17%
  |                                                                                                                         
  |====================                                                                                               |  18%
  |                                                                                                                         
  |=====================                                                                                              |  18%
  |                                                                                                                         
  |=====================                                                                                              |  19%
  |                                                                                                                         
  |======================                                                                                             |  19%
  |                                                                                                                         
  |======================                                                                                             |  20%
  |                                                                                                                         
  |=======================                                                                                            |  20%
  |                                                                                                                         
  |========================                                                                                           |  20%
  |                                                                                                                         
  |========================                                                                                           |  21%
  |                                                                                                                         
  |=========================                                                                                          |  21%
  |                                                                                                                         
  |=========================                                                                                          |  22%
  |                                                                                                                         
  |==========================                                                                                         |  22%
  |                                                                                                                         
  |==========================                                                                                         |  23%
  |                                                                                                                         
  |===========================                                                                                        |  23%
  |                                                                                                                         
  |===========================                                                                                        |  24%
  |                                                                                                                         
  |============================                                                                                       |  24%
  |                                                                                                                         
  |============================                                                                                       |  25%
  |                                                                                                                         
  |=============================                                                                                      |  25%
  |                                                                                                                         
  |=============================                                                                                      |  26%
  |                                                                                                                         
  |==============================                                                                                     |  26%
  |                                                                                                                         
  |===============================                                                                                    |  27%
  |                                                                                                                         
  |================================                                                                                   |  27%
  |                                                                                                                         
  |================================                                                                                   |  28%
  |                                                                                                                         
  |=================================                                                                                  |  28%
  |                                                                                                                         
  |=================================                                                                                  |  29%
  |                                                                                                                         
  |==================================                                                                                 |  29%
  |                                                                                                                         
  |==================================                                                                                 |  30%
  |                                                                                                                         
  |===================================                                                                                |  30%
  |                                                                                                                         
  |===================================                                                                                |  31%
  |                                                                                                                         
  |====================================                                                                               |  31%
  |                                                                                                                         
  |====================================                                                                               |  32%
  |                                                                                                                         
  |=====================================                                                                              |  32%
  |                                                                                                                         
  |=====================================                                                                              |  33%
  |                                                                                                                         
  |======================================                                                                             |  33%
  |                                                                                                                         
  |=======================================                                                                            |  34%
  |                                                                                                                         
  |========================================                                                                           |  34%
  |                                                                                                                         
  |========================================                                                                           |  35%
  |                                                                                                                         
  |=========================================                                                                          |  35%
  |                                                                                                                         
  |=========================================                                                                          |  36%
  |                                                                                                                         
  |==========================================                                                                         |  36%
  |                                                                                                                         
  |==========================================                                                                         |  37%
  |                                                                                                                         
  |===========================================                                                                        |  37%
  |                                                                                                                         
  |===========================================                                                                        |  38%
  |                                                                                                                         
  |============================================                                                                       |  38%
  |                                                                                                                         
  |============================================                                                                       |  39%
  |                                                                                                                         
  |=============================================                                                                      |  39%
  |                                                                                                                         
  |=============================================                                                                      |  40%
  |                                                                                                                         
  |==============================================                                                                     |  40%
  |                                                                                                                         
  |===============================================                                                                    |  40%
  |                                                                                                                         
  |===============================================                                                                    |  41%
  |                                                                                                                         
  |================================================                                                                   |  41%
  |                                                                                                                         
  |================================================                                                                   |  42%
  |                                                                                                                         
  |=================================================                                                                  |  42%
  |                                                                                                                         
  |=================================================                                                                  |  43%
  |                                                                                                                         
  |==================================================                                                                 |  43%
  |                                                                                                                         
  |==================================================                                                                 |  44%
  |                                                                                                                         
  |===================================================                                                                |  44%
  |                                                                                                                         
  |===================================================                                                                |  45%
  |                                                                                                                         
  |====================================================                                                               |  45%
  |                                                                                                                         
  |====================================================                                                               |  46%
  |                                                                                                                         
  |=====================================================                                                              |  46%
  |                                                                                                                         
  |======================================================                                                             |  47%
  |                                                                                                                         
  |=======================================================                                                            |  47%
  |                                                                                                                         
  |=======================================================                                                            |  48%
  |                                                                                                                         
  |========================================================                                                           |  48%
  |                                                                                                                         
  |========================================================                                                           |  49%
  |                                                                                                                         
  |=========================================================                                                          |  49%
  |                                                                                                                         
  |=========================================================                                                          |  50%
  |                                                                                                                         
  |==========================================================                                                         |  50%
  |                                                                                                                         
  |==========================================================                                                         |  51%
  |                                                                                                                         
  |===========================================================                                                        |  51%
  |                                                                                                                         
  |===========================================================                                                        |  52%
  |                                                                                                                         
  |============================================================                                                       |  52%
  |                                                                                                                         
  |============================================================                                                       |  53%
  |                                                                                                                         
  |=============================================================                                                      |  53%
  |                                                                                                                         
  |==============================================================                                                     |  54%
  |                                                                                                                         
  |===============================================================                                                    |  54%
  |                                                                                                                         
  |===============================================================                                                    |  55%
  |                                                                                                                         
  |================================================================                                                   |  55%
  |                                                                                                                         
  |================================================================                                                   |  56%
  |                                                                                                                         
  |=================================================================                                                  |  56%
  |                                                                                                                         
  |=================================================================                                                  |  57%
  |                                                                                                                         
  |==================================================================                                                 |  57%
  |                                                                                                                         
  |==================================================================                                                 |  58%
  |                                                                                                                         
  |===================================================================                                                |  58%
  |                                                                                                                         
  |===================================================================                                                |  59%
  |                                                                                                                         
  |====================================================================                                               |  59%
  |                                                                                                                         
  |====================================================================                                               |  60%
  |                                                                                                                         
  |=====================================================================                                              |  60%
  |                                                                                                                         
  |======================================================================                                             |  60%
  |                                                                                                                         
  |======================================================================                                             |  61%
  |                                                                                                                         
  |=======================================================================                                            |  61%
  |                                                                                                                         
  |=======================================================================                                            |  62%
  |                                                                                                                         
  |========================================================================                                           |  62%
  |                                                                                                                         
  |========================================================================                                           |  63%
  |                                                                                                                         
  |=========================================================================                                          |  63%
  |                                                                                                                         
  |=========================================================================                                          |  64%
  |                                                                                                                         
  |==========================================================================                                         |  64%
  |                                                                                                                         
  |==========================================================================                                         |  65%
  |                                                                                                                         
  |===========================================================================                                        |  65%
  |                                                                                                                         
  |===========================================================================                                        |  66%
  |                                                                                                                         
  |============================================================================                                       |  66%
  |                                                                                                                         
  |=============================================================================                                      |  67%
  |                                                                                                                         
  |==============================================================================                                     |  67%
  |                                                                                                                         
  |==============================================================================                                     |  68%
  |                                                                                                                         
  |===============================================================================                                    |  68%
  |                                                                                                                         
  |===============================================================================                                    |  69%
  |                                                                                                                         
  |================================================================================                                   |  69%
  |                                                                                                                         
  |================================================================================                                   |  70%
  |                                                                                                                         
  |=================================================================================                                  |  70%
  |                                                                                                                         
  |=================================================================================                                  |  71%
  |                                                                                                                         
  |==================================================================================                                 |  71%
  |                                                                                                                         
  |==================================================================================                                 |  72%
  |                                                                                                                         
  |===================================================================================                                |  72%
  |                                                                                                                         
  |===================================================================================                                |  73%
  |                                                                                                                         
  |====================================================================================                               |  73%
  |                                                                                                                         
  |=====================================================================================                              |  74%
  |                                                                                                                         
  |======================================================================================                             |  74%
  |                                                                                                                         
  |======================================================================================                             |  75%
  |                                                                                                                         
  |=======================================================================================                            |  75%
  |                                                                                                                         
  |=======================================================================================                            |  76%
  |                                                                                                                         
  |========================================================================================                           |  76%
  |                                                                                                                         
  |========================================================================================                           |  77%
  |                                                                                                                         
  |=========================================================================================                          |  77%
  |                                                                                                                         
  |=========================================================================================                          |  78%
  |                                                                                                                         
  |==========================================================================================                         |  78%
  |                                                                                                                         
  |==========================================================================================                         |  79%
  |                                                                                                                         
  |===========================================================================================                        |  79%
  |                                                                                                                         
  |===========================================================================================                        |  80%
  |                                                                                                                         
  |============================================================================================                       |  80%
  |                                                                                                                         
  |=============================================================================================                      |  80%
  |                                                                                                                         
  |=============================================================================================                      |  81%
  |                                                                                                                         
  |==============================================================================================                     |  81%
  |                                                                                                                         
  |==============================================================================================                     |  82%
  |                                                                                                                         
  |===============================================================================================                    |  82%
  |                                                                                                                         
  |===============================================================================================                    |  83%
  |                                                                                                                         
  |================================================================================================                   |  83%
  |                                                                                                                         
  |================================================================================================                   |  84%
  |                                                                                                                         
  |=================================================================================================                  |  84%
  |                                                                                                                         
  |=================================================================================================                  |  85%
  |                                                                                                                         
  |==================================================================================================                 |  85%
  |                                                                                                                         
  |==================================================================================================                 |  86%
  |                                                                                                                         
  |===================================================================================================                |  86%
  |                                                                                                                         
  |====================================================================================================               |  87%
  |                                                                                                                         
  |=====================================================================================================              |  87%
  |                                                                                                                         
  |=====================================================================================================              |  88%
  |                                                                                                                         
  |======================================================================================================             |  88%
  |                                                                                                                         
  |======================================================================================================             |  89%
  |                                                                                                                         
  |=======================================================================================================            |  89%
  |                                                                                                                         
  |=======================================================================================================            |  90%
  |                                                                                                                         
  |========================================================================================================           |  90%
  |                                                                                                                         
  |========================================================================================================           |  91%
  |                                                                                                                         
  |=========================================================================================================          |  91%
  |                                                                                                                         
  |=========================================================================================================          |  92%
  |                                                                                                                         
  |==========================================================================================================         |  92%
  |                                                                                                                         
  |==========================================================================================================         |  93%
  |                                                                                                                         
  |===========================================================================================================        |  93%
  |                                                                                                                         
  |============================================================================================================       |  94%
  |                                                                                                                         
  |=============================================================================================================      |  94%
  |                                                                                                                         
  |=============================================================================================================      |  95%
  |                                                                                                                         
  |==============================================================================================================     |  95%
  |                                                                                                                         
  |==============================================================================================================     |  96%
  |                                                                                                                         
  |===============================================================================================================    |  96%
  |                                                                                                                         
  |===============================================================================================================    |  97%
  |                                                                                                                         
  |================================================================================================================   |  97%
  |                                                                                                                         
  |================================================================================================================   |  98%
  |                                                                                                                         
  |=================================================================================================================  |  98%
  |                                                                                                                         
  |=================================================================================================================  |  99%
  |                                                                                                                         
  |================================================================================================================== |  99%
  |                                                                                                                         
  |================================================================================================================== | 100%
  |                                                                                                                         
  |===================================================================================================================| 100%
Centering data matrix

  |                                                                                                                         
  |                                                                                                                   |   0%
  |                                                                                                                         
  |=============================                                                                                      |  25%
  |                                                                                                                         
  |==========================================================                                                         |  50%
  |                                                                                                                         
  |======================================================================================                             |  75%
  |                                                                                                                         
  |===================================================================================================================| 100%
Set default assay to SCT

Tada! Now lets do some plotting and clustering!

linear dimensional reduction. Default are based on VariableFeatures, but can be changed

seurat.object <- RunPCA(seurat.object, features = VariableFeatures(object = seurat.object), verbose = FALSE)
VizDimLoadings(seurat.object, dims = 1:6, nfeatures = 10, reduction = "pca", ncol = 2)

DimPlot colored by orig.ident

DimPlot(seurat.object, reduction = "pca", group.by = "orig.ident")

source("msAggr_AnalysisCode/PercentVariance.R")
ElbowPlot(seurat.object, ndims = 50)
percent.variance(seurat.object@reductions$pca@stdev)

Variance described by dimensionality

Number of PCs describing X% of variance

ElbowPlot(seurat.object, ndims = 50)

percent.variance(seurat.object@reductions$pca@stdev)

Dim33

Cover 95% variability

tot.var <- percent.variance(seurat.object@reductions$pca@stdev, plot.var = FALSE, return.val = TRUE)
paste0("Num pcs for 80% variance:", length(which(cumsum(tot.var) <= 80)))
[1] "Num pcs for 80% variance:11"
paste0("Num pcs for 85% variance:", length(which(cumsum(tot.var) <= 85)))
[1] "Num pcs for 85% variance:15"
paste0("Num pcs for 90% variance:", length(which(cumsum(tot.var) <= 90)))
[1] "Num pcs for 90% variance:22"
paste0("Num pcs for 95% variance:", length(which(cumsum(tot.var) <= 95)))
[1] "Num pcs for 95% variance:33"
ndim <- 33

Plotting…

seurat.object <- RunUMAP(seurat.object, dims = 1:33)
16:50:51 UMAP embedding parameters a = 0.9922 b = 1.112
16:50:51 Read 40815 rows and found 33 numeric columns
16:50:51 Using Annoy for neighbor search, n_neighbors = 30
16:50:51 Building Annoy index with metric = cosine, n_trees = 50
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
16:50:56 Writing NN index file to temp file /var/folders/4f/fwrj6fnn1dn4g8wsf0zv563hjsvl24/T//RtmpLSZxR7/file16fc1e0cf629
16:50:56 Searching Annoy index using 1 thread, search_k = 3000
16:51:07 Annoy recall = 100%
16:51:08 Commencing smooth kNN distance calibration using 1 thread
16:51:10 Initializing from normalized Laplacian + noise
16:51:12 Commencing optimization for 200 epochs, with 1697286 positive edges
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
16:51:32 Optimization finished
seurat.object <- FindNeighbors(seurat.object, dims = 1:33)
Computing nearest neighbor graph
Computing SNN
for(x in c(0.5, 1, 1.5, 2, 2.5)){
    seurat.object <- FindClusters(seurat.object, resolution = x)
}
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 40815
Number of edges: 1349968

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.9113
Number of communities: 17
Elapsed time: 8 seconds
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 40815
Number of edges: 1349968

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.8778
Number of communities: 27
Elapsed time: 8 seconds
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 40815
Number of edges: 1349968

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.8566
Number of communities: 36
Elapsed time: 8 seconds
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 40815
Number of edges: 1349968

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.8392
Number of communities: 46
Elapsed time: 8 seconds
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 40815
Number of edges: 1349968

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.8250
Number of communities: 48
Elapsed time: 7 seconds
for (meta.col in colnames(seurat.object@meta.data)){
    if(grepl(pattern = ("SCT_snn_res"), x = meta.col)==TRUE){
        myplot <- DimPlot(seurat.object, 
                                            group.by = meta.col,
                                            reduction = "umap", 
                                            cols = colorRamps::primary.colors(n = length(levels(seurat.object@meta.data[[meta.col]])))
                                            ) + 
            ggtitle(paste0("msAggr sct dim31 res", gsub("SCT_snn_res", "", meta.col) ))
        plot(myplot)
    }
}

Evaluate cluster stability

Must ensure we have the right cluster stability, that is, cells that start in the same cluster tend to stay in the same cluster. If your data is over-clustered, cells will bounce between groups.

Following [this tutorial by Matt O.].https://towardsdatascience.com/10-tips-for-choosing-the-optimal-number-of-clusters-277e93d72d92.

Clustree

Previously my favourite has been Clustree, which gives a nice visual NB: For some reason clustree::clustree() didn’t work, whereas library(clustree) followed by clustree() did.

saveRDS(seurat.object, file = "msAggr_sctdim33.rds")

Basically no stability difference compared to before.

Dim22

Cover 90% variability

clustree(seurat.object, prefix = "SCT_snn_res.", node_colour = "sc3_stability") + 
    scale_color_continuous(low = 'red3', high = 'white')

ndim <- 22

Plotting…

seurat.object <- RunUMAP(seurat.object, dims = 1:ndim)
09:11:45 UMAP embedding parameters a = 0.9922 b = 1.112
09:11:45 Read 40815 rows and found 22 numeric columns
09:11:45 Using Annoy for neighbor search, n_neighbors = 30
09:11:45 Building Annoy index with metric = cosine, n_trees = 50
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
09:11:49 Writing NN index file to temp file /var/folders/4f/fwrj6fnn1dn4g8wsf0zv563hjsvl24/T//RtmpLSZxR7/file16fc360e68c6
09:11:49 Searching Annoy index using 1 thread, search_k = 3000
09:11:59 Annoy recall = 100%
09:12:00 Commencing smooth kNN distance calibration using 1 thread
09:12:02 Initializing from normalized Laplacian + noise
09:12:05 Commencing optimization for 200 epochs, with 1662108 positive edges
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
*****************************
for (meta.col in colnames(seurat.object@meta.data)){
    if(grepl(pattern = ("SCT_snn_res"), x = meta.col)==TRUE){
        myplot <- DimPlot(seurat.object, 
                                            group.by = meta.col,
                                            reduction = "umap", 
                                            cols = colorRamps::primary.colors(n = length(levels(seurat.object@meta.data[[meta.col]])))
                                            ) + 
            ggtitle(paste0("msAggr sct dim", ndim, "res", gsub("SCT_snn_res", "", meta.col) ))
        plot(myplot)
    }
}

Evaluate cluster stability

Must ensure we have the right cluster stability, that is, cells that start in the same cluster tend to stay in the same cluster. If your data is over-clustered, cells will bounce between groups.

Following [this tutorial by Matt O.].https://towardsdatascience.com/10-tips-for-choosing-the-optimal-number-of-clusters-277e93d72d92.

Clustree

Previously my favourite has been Clustree, which gives a nice visual NB: For some reason clustree::clustree() didn’t work, whereas library(clustree) followed by clustree() did.

saveRDS(seurat.object, file = paste0("msAggr_sctdim", ndim, ".rds"))

Hard to tell if dim33 or dim22 is better, but if feels like dim22 has less cross-over between res0.5 and res1.0. Will continue at dim22, and see how things change between res0.5 and res1.0

clustree(seurat.object, prefix = "SCT_snn_res.", node_colour = "sc3_stability") + 
    scale_color_continuous(low = 'red3', high = 'white')

MitoC in

for (x in c("LSKm2", "CMPm2", "MEPm", "GMPm")){
    h = hist(seurat.object@meta.data[seurat.object@meta.data$orig.ident == x, 'percent.mt'], breaks = 30, plot = FALSE)
    h$density = h$counts/sum(h$counts)*100
    plot(h,freq=FALSE, main =  paste(x, "percent mitoC"), xlab = "percent mitoC", ylab = "Frequency")
}
LS0tCnRpdGxlOiAibXNBZ2dyX3NjdHJhbnNmb3JtLXYxIgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKLS0tCgpDcmVhdGluZyBuZXcgcGlwZWxpbmUgdXNpbmcgc2V1cmF0IHY0LjAuMiBhdmFpbGFibGUgMjAyMS4wNi4wOAoKTG9hZCBsaWJyYXJpZXMgcmVxdWlyZWQgZm9yIFNldXJhdHY0CgpgYGB7ciBzZXR1cH0Ka25pdHI6Om9wdHNfa25pdCRzZXQocm9vdC5kaXIgPSAifi9EZXNrdG9wLzEwWEdlbm9taWNzRGF0YS9tc0FnZ3Jfc2NSTkFTZXEvIikKbGlicmFyeShkcGx5cikKbGlicmFyeShTZXVyYXQpCmxpYnJhcnkocGF0Y2h3b3JrKQpsaWJyYXJ5KGdncGxvdDIpCmxpYnJhcnkoY2x1c3RyZWUpCmBgYApzdG9yZSBzZXNzaW9uIGluZm8KYGBge3J9CnNpbmsoIm1zQWdncl9zY3QtdjEuMjAyMTA2MTUiKQpzZXNzaW9uSW5mbygpCnNpbmsoKQpgYGAKCiMgQSBub3RlIGFib3V0IHVzaW5nIFNDVHJhbnNmb3JtIHZlcnN1cyBgU2NhbGVEYXRhYApodHRwczovL2Jpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvMy4xMC93b3JrZmxvd3MvdmlnbmV0dGVzL3NpbXBsZVNpbmdsZUNlbGwvaW5zdC9kb2MvYmF0Y2guaHRtbCM2Ml9mb3JfZ2VuZS1iYXNlZF9hbmFseXNlcwo+WW91IGNhbiBhbHNvIG5vcm1hbGl6ZSBhbmQgc2NhbGUgZGF0YSBmb3IgdGhlIFJOQSBhc3NheS4gVGhlcmUgYXJlIG51bWVyb3VzIHJlc291cmNlcyBvbiB0aGlzLCBidXQgQWFyb24gTHVuIGRlc2NyaWJlcyB3aHkgdGhlIG9yaWdpbmFsIGxvZy1ub3JtYWxpemVkIHZhbHVlcyBzaG91bGQgYmUgdXNlZCBmb3IgREUgYW5kIHZpc3VhbGl6YXRpb25zIG9mIGV4cHJlc3Npb24gcXVpdGUgd2VsbCBoZXJlOgo+Cj5Gb3IgZ2VuZS1iYXNlZCBwcm9jZWR1cmVzIGxpa2UgZGlmZmVyZW50aWFsIGV4cHJlc3Npb24gKERFKSBhbmFseXNlcyBvciBnZW5lIG5ldHdvcmsgY29uc3RydWN0aW9uLCBpdCBpcyBkZXNpcmFibGUgdG8gdXNlIHRoZSBvcmlnaW5hbCBsb2ctZXhwcmVzc2lvbiB2YWx1ZXMgb3IgY291bnRzLiBUaGUgY29ycmVjdGVkIHZhbHVlcyBhcmUgb25seSB1c2VkIHRvIG9idGFpbiBjZWxsLWxldmVsIHJlc3VsdHMgc3VjaCBhcyBjbHVzdGVycyBvciB0cmFqZWN0b3JpZXMuIEJhdGNoIGVmZmVjdHMgYXJlIGhhbmRsZWQgZXhwbGljaXRseSB1c2luZyBibG9ja2luZyB0ZXJtcyBvciB2aWEgYSBtZXRhLWFuYWx5c2lzIGFjcm9zcyBiYXRjaGVzLiBXZSBkbyBub3QgdXNlIHRoZSBjb3JyZWN0ZWQgdmFsdWVzIGRpcmVjdGx5IGluIGdlbmUtYmFzZWQgYW5hbHlzZXMsIGZvciB2YXJpb3VzIHJlYXNvbnM6Cj4KPkl0IGlzIHVzdWFsbHkgaW5hcHByb3ByaWF0ZSB0byBwZXJmb3JtIERFIGFuYWx5c2VzIG9uIGJhdGNoLWNvcnJlY3RlZCB2YWx1ZXMsIGR1ZSB0byB0aGUgZmFpbHVyZSB0byBtb2RlbCB0aGUgdW5jZXJ0YWludHkgb2YgdGhlIGNvcnJlY3Rpb24uIFRoaXMgdXN1YWxseSByZXN1bHRzIGluIGxvc3Mgb2YgdHlwZSBJIGVycm9yIGNvbnRyb2wsIGkuZS4sIG1vcmUgZmFsc2UgcG9zaXRpdmVzIHRoYW4gZXhwZWN0ZWQuCj4KPlRoZSBjb3JyZWN0aW9uIGRvZXMgbm90IHByZXNlcnZlIHRoZSBtZWFuLXZhcmlhbmNlIHJlbGF0aW9uc2hpcC4gQXBwbGljYXRpb25zIG9mIGNvbW1vbiBERSBtZXRob2RzIGxpa2UgZWRnZVIgb3IgbGltbWEgYXJlIHVubGlrZWx5IHRvIGJlIHZhbGlkLgo+Cj5CYXRjaCBjb3JyZWN0aW9uIG1heSAoY29ycmVjdGx5KSByZW1vdmUgYmlvbG9naWNhbCBkaWZmZXJlbmNlcyBiZXR3ZWVuIGJhdGNoZXMgaW4gdGhlIGNvdXJzZSBvZiBtYXBwaW5nIGFsbCBjZWxscyBvbnRvIGEgY29tbW9uIGNvb3JkaW5hdGUgc3lzdGVtLiBSZXR1cm5pbmcgdG8gdGhlIHVuY29ycmVjdGVkIGV4cHJlc3Npb24gdmFsdWVzIHByb3ZpZGVzIGFuIG9wcG9ydHVuaXR5IGZvciBkZXRlY3Rpbmcgc3VjaCBkaWZmZXJlbmNlcyBpZiB0aGV5IGFyZSBvZiBpbnRlcmVzdC4gQ29udmVyc2VseSwgaWYgdGhlIGJhdGNoIGNvcnJlY3Rpb24gbWFkZSBhIG1pc3Rha2UsIHRoZSB1c2Ugb2YgdGhlIHVuY29ycmVjdGVkIGV4cHJlc3Npb24gdmFsdWVzIHByb3ZpZGVzIGFuIGltcG9ydGFudCBzYW5pdHkgY2hlY2suCj4KPkluIGFkZGl0aW9uLCB0aGUgbm9ybWFsaXplZCB2YWx1ZXMgaW4gU0NUIGFuZCBpbnRlZ3JhdGVkIGFzc2F5cyBkb24ndCBuZWNlc3NhcnkgY29ycmVzcG9uZCB0byBwZXItZ2VuZSBleHByZXNzaW9uIHZhbHVlcyBhbnl3YXksIHJhdGhlciBjb250YWluaW5nIHJlc2lkdWFscyAoaW4gdGhlIGNhc2Ugb2YgdGhlIHNjYWxlLmRhdGEgc2xvdCBmb3IgZWFjaCkuCgoKTG9hZCA0IHBvcHVsYXRpb25zIGludG8gc2luZ2xlIHNldXJhdCBvYmplY3QKCgpTRVQgU0VFRD8/Pz8/ISEhISEKCiMgU2V0IGdsb2JhbCB2YXJpYWJsZXMKYGBge3J9CnByb2plY3ROYW1lIDwtICJtc0FnZ3Jfc2N0IgpqYWNrc3RyYXcuZGltIDwtIDQwCmBgYAoKCgpgYGB7cn0Kc291cmNlKCJtc0FnZ3JfQW5hbHlzaXNDb2RlL3JlYWRfMTBYR2Vub21pY3NfZGF0YS5SIikKCmBgYAoKCmBgYHtyfQpzZXR3ZCgiLi4vY2VsbFJhbmdlci8iKSAjIHRlbXBvcmFyaWx5IGNoYW5naW5nIHdkIG9ubHkgd29ya3MgaWYgeW91IHJ1biB0aGUgZW50aXJlIGNodW5rIGF0IG9uY2UKZGF0YV9maWxlLmxpc3QgPC0gcmVhZF8xMFhHZW5vbWljc19kYXRhKHNhbXBsZS5saXN0ID0gYygiTFNLbTIiLCAiQ01QbTIiLCAiTUVQbSIsICJHTVBtIikpCm9iamVjdC5kYXRhIDwtUmVhZDEwWChkYXRhX2ZpbGUubGlzdCkKYGBgCgoKCmBgYHtyfQpzZXVyYXQub2JqZWN0PC0gQ3JlYXRlU2V1cmF0T2JqZWN0KGNvdW50cyA9IG9iamVjdC5kYXRhLCBtaW4uY2VsbHMgPSAzLCBtaW4uZ2VuZXMgPSAyMDAsIHByb2plY3QgPSBwcm9qZWN0TmFtZSkKYGBgCgpDbGVhbiB1cCB0byBmcmVlIG1lbW9yeQoKYGBge3J9CnJlbW92ZShzZXVyYXQub2JqZWN0LmRhdGEpCmBgYAoKCkFkZCBtaXRvY2hvbmRyaWFsIG1ldGFkYXRhIGFuZCBwbG90IHNvbWUgYmFzaWMgZmVhdHVyZXMKYGBge3J9CnNldXJhdC5vYmplY3RbWyJwZXJjZW50Lm10Il1dIDwtIFBlcmNlbnRhZ2VGZWF0dXJlU2V0KHNldXJhdC5vYmplY3QsIHBhdHRlcm4gPSAiXm10LSIpClZsblBsb3Qoc2V1cmF0Lm9iamVjdCwgZmVhdHVyZXMgPSBjKCJuRmVhdHVyZV9STkEiLCAibkNvdW50X1JOQSIsICJwZXJjZW50Lm10IiksIG5jb2wgPSAzLCBwdC5zaXplID0gMCwgZmlsbC5ieSA9ICdvcmlnLmlkZW50JywgKQpgYGAKCgpgYGB7ciBmaWcud2lkdGg9NCwgZmlnLmhlaWdodD0yfQpwbG90MSA8LSBGZWF0dXJlU2NhdHRlcihzZXVyYXQub2JqZWN0LCBmZWF0dXJlMSA9ICJuQ291bnRfUk5BIiwgZmVhdHVyZTIgPSAicGVyY2VudC5tdCIsIGdyb3VwLmJ5ID0gIm9yaWcuaWRlbnQiLCBwdC5zaXplID0gMC4wMSkKcGxvdDIgPC0gRmVhdHVyZVNjYXR0ZXIoc2V1cmF0Lm9iamVjdCwgZmVhdHVyZTEgPSAibkNvdW50X1JOQSIsIGZlYXR1cmUyID0gIm5GZWF0dXJlX1JOQSIsIGdyb3VwLmJ5ID0gIm9yaWcuaWRlbnQiLCBwdC5zaXplID0gMC4wMSkKcGxvdDEgKyBwbG90MgpgYGAKcmVtb3ZlIGxvdyBxdWFsaXR5IGNlbGxzICoqCnJlcXVpcmU6IG5GZWF0dXJlX1JOQSBiZXR3ZWVuIDIwMCBhbmQgNDAwMCAoaW5jbHVzaXZlKQotLXJlcXVpcmU6IHBlcmNlbnQubXQgPDUtLT8/PwoKXCoqIEluIHByZXZpb3VzIHJ1biBpdCdzIHBvc3NpYmxlIHRoYXQgd2UnbGwgbmVlZCB0byBsaW1pdCBwZXJjZW50Lm10IGJlY2F1c2Ugb2YgTUVQbSB3ZWlyZG5lc3MuIFdpbGwgaGF2ZSB0byBzZWUgaWYgdGhlIGFkZGl0aW9uYWwgMjAwMCBNRVBtIGNlbGxzIGZvdW5kIGJ5IGNlbGxyYW5nZXIgdjYKCmBgYHtyfQpwcmludChwYXN0ZSgib3JpZ2luYWwgb2JqZWN0OiIsIG5yb3coc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEpLCAiY2VsbHMiLCBzZXAgPSAiICIpKQpzZXVyYXQub2JqZWN0IDwtIHN1YnNldChzZXVyYXQub2JqZWN0LCAKCQkJCQkJCQkJCQkJc3Vic2V0ID0gbkZlYXR1cmVfUk5BID49MjAwICYgCgkJCQkJCQkJCQkJCQluRmVhdHVyZV9STkEgPD0gNDAwMAoJCQkJCQkJCQkJCQkpCnByaW50KHBhc3RlKCJuZXcgb2JqZWN0OiIsIG5yb3coc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEpLCAiY2VsbHMiLCBzZXAgPSAiICIpKQpzZXVyYXQub2JqZWN0IDwtIFBlcmNlbnRhZ2VGZWF0dXJlU2V0KHNldXJhdC5vYmplY3QsIHBhdHRlcm4gPSAnXm10LScsIGNvbC5uYW1lID0gInBlcmNlbnQubXQiKQpgYGAKCgoKIyMgU0NUcmFuc2Zvcm0KCkFzIHByZXZpb3VzbHkgbWVudGlvbmVkLCBgU0NUcmFuc2Zvcm0oKWAgaXMgYmVzdCBmb3IgYmF0Y2ggY29ycmVjdGlvbiwgd2hlcmVhcyBgTm9ybWFsaXplRGF0YSgpYCwgYFNjYWxlRGF0YSgpYCwgYW5kIGBGaW5kVmFyaWFibGVGZWF0dXJlcygpYCBhcmUgYmVzdCBmb3IgREdFLiBJJ20gc3RydWdnbGluZyB0byBtYWtlIHN0YWJsZSBjbHVzdGVycywgc28gSSdtIGp1bXBpbmcgYWhlYWQgdG8gYFNDVHJhbnNmb3JtKClgLiAgCkl0IGRvZXMgYXBwZWFyIHRoYXQgYFNDVHJhbnNmb3JtKClgIGRhdGEgYXJlIHN0b3JlZCBpbiB0aGUgU0NUIGFzc2F5IHNsb3QsIHdoaWNoIGRvZXMgbm90IG92ZXJ3cml0ZSBvdGhlciBub3JtYWxpemF0aW9uIHNsb3RzLiBJdCB3aWxsIGJlY29tZSB0aGUgZGVmYXVsdCBzbG90IG9uY2UgcG9wdWxhdGVkLgoKYGBge3J9CnNldXJhdC5vYmplY3QgPC0gU0NUcmFuc2Zvcm0oc2V1cmF0Lm9iamVjdCwgCgkJCQkJCQkJCQkJCQkJIG1ldGhvZCA9ICJnbG1HYW1Qb2kiLCAKCQkJCQkJCQkJCQkJCQkgdmFycy50by5yZWdyZXNzID0gYygibkNvdW50X1JOQSIsICJuRmVhdHVyZV9STkEiKQoJCQkJCQkJCQkJCQkJCSApICMgbWV0aG9kID0gImdsbUdhbVBvaSIgc3BlZWRzIHByb2Nlc3MKYGBgCgpUYWRhISBOb3cgbGV0cyBkbyBzb21lIHBsb3R0aW5nIGFuZCBjbHVzdGVyaW5nIQoKbGluZWFyIGRpbWVuc2lvbmFsIHJlZHVjdGlvbi4gRGVmYXVsdCBhcmUgYmFzZWQgb24gVmFyaWFibGVGZWF0dXJlcywgYnV0IGNhbiBiZSBjaGFuZ2VkCgpgYGB7cn0Kc2V1cmF0Lm9iamVjdCA8LSBSdW5QQ0Eoc2V1cmF0Lm9iamVjdCwgZmVhdHVyZXMgPSBWYXJpYWJsZUZlYXR1cmVzKG9iamVjdCA9IHNldXJhdC5vYmplY3QpLCB2ZXJib3NlID0gRkFMU0UpCmBgYAoKCmBgYHtyIGZpZy53aWR0aD00LCBmaWcuaGVpZ2h0PTR9ClZpekRpbUxvYWRpbmdzKHNldXJhdC5vYmplY3QsIGRpbXMgPSAxOjYsIG5mZWF0dXJlcyA9IDEwLCByZWR1Y3Rpb24gPSAicGNhIiwgbmNvbCA9IDIpCmBgYAoKRGltUGxvdCBjb2xvcmVkIGJ5IG9yaWcuaWRlbnQKYGBge3J9CkRpbVBsb3Qoc2V1cmF0Lm9iamVjdCwgcmVkdWN0aW9uID0gInBjYSIsIGdyb3VwLmJ5ID0gIm9yaWcuaWRlbnQiKQpgYGAKCmBgYHtyfQpzb3VyY2UoIm1zQWdncl9BbmFseXNpc0NvZGUvUGVyY2VudFZhcmlhbmNlLlIiKQpgYGAKCmBgYHtyLCBmaWd1cmVzLXNpZGUsIGZpZy5zaG93PSdob2xkJywgb3V0LndpZHRoPSI1MCUifQpFbGJvd1Bsb3Qoc2V1cmF0Lm9iamVjdCwgbmRpbXMgPSA1MCkKcGVyY2VudC52YXJpYW5jZShzZXVyYXQub2JqZWN0QHJlZHVjdGlvbnMkcGNhQHN0ZGV2KQpgYGAKIyBWYXJpYW5jZSBkZXNjcmliZWQgYnkgZGltZW5zaW9uYWxpdHkKTnVtYmVyIG9mIFBDcyBkZXNjcmliaW5nIFglIG9mIHZhcmlhbmNlCgpgYGB7cn0KdG90LnZhciA8LSBwZXJjZW50LnZhcmlhbmNlKHNldXJhdC5vYmplY3RAcmVkdWN0aW9ucyRwY2FAc3RkZXYsIHBsb3QudmFyID0gRkFMU0UsIHJldHVybi52YWwgPSBUUlVFKQpwYXN0ZTAoIk51bSBwY3MgZm9yIDgwJSB2YXJpYW5jZToiLCBsZW5ndGgod2hpY2goY3Vtc3VtKHRvdC52YXIpIDw9IDgwKSkpCnBhc3RlMCgiTnVtIHBjcyBmb3IgODUlIHZhcmlhbmNlOiIsIGxlbmd0aCh3aGljaChjdW1zdW0odG90LnZhcikgPD0gODUpKSkKcGFzdGUwKCJOdW0gcGNzIGZvciA5MCUgdmFyaWFuY2U6IiwgbGVuZ3RoKHdoaWNoKGN1bXN1bSh0b3QudmFyKSA8PSA5MCkpKQpwYXN0ZTAoIk51bSBwY3MgZm9yIDk1JSB2YXJpYW5jZToiLCBsZW5ndGgod2hpY2goY3Vtc3VtKHRvdC52YXIpIDw9IDk1KSkpCgpgYGAKCiMgRGltMzMKCkNvdmVyIDk1JSB2YXJpYWJpbGl0eQoKYGBge3J9Cm5kaW0gPC0gMzMKYGBgCgoKYGBge3J9CnNldXJhdC5vYmplY3QgPC0gUnVuVU1BUChzZXVyYXQub2JqZWN0LCBkaW1zID0gMTpuZGltKQpzZXVyYXQub2JqZWN0IDwtIEZpbmROZWlnaGJvcnMoc2V1cmF0Lm9iamVjdCwgZGltcyA9IDE6bmRpbSkKZm9yKHggaW4gYygwLjUsIDEsIDEuNSwgMiwgMi41KSl7CglzZXVyYXQub2JqZWN0IDwtIEZpbmRDbHVzdGVycyhzZXVyYXQub2JqZWN0LCByZXNvbHV0aW9uID0geCkKfQpgYGAKUGxvdHRpbmcuLi4KCgpgYGB7cn0KZm9yIChtZXRhLmNvbCBpbiBjb2xuYW1lcyhzZXVyYXQub2JqZWN0QG1ldGEuZGF0YSkpewoJaWYoZ3JlcGwocGF0dGVybiA9ICgiU0NUX3Nubl9yZXMiKSwgeCA9IG1ldGEuY29sKT09VFJVRSl7CgkJbXlwbG90IDwtIERpbVBsb3Qoc2V1cmF0Lm9iamVjdCwgCgkJCQkJCQkJCQkJZ3JvdXAuYnkgPSBtZXRhLmNvbCwKCQkJCQkJCQkJCQlyZWR1Y3Rpb24gPSAidW1hcCIsIAoJCQkJCQkJCQkJCWNvbHMgPSBjb2xvclJhbXBzOjpwcmltYXJ5LmNvbG9ycyhuID0gbGVuZ3RoKGxldmVscyhzZXVyYXQub2JqZWN0QG1ldGEuZGF0YVtbbWV0YS5jb2xdXSkpKQoJCQkJCQkJCQkJCSkgKyAKCQkJZ2d0aXRsZShwYXN0ZTAoIm1zQWdnciBzY3QgZGltMzEgcmVzIiwgZ3N1YigiU0NUX3Nubl9yZXMiLCAiIiwgbWV0YS5jb2wpICkpCgkJcGxvdChteXBsb3QpCgl9Cn0KYGBgCgoKCgpgYGB7cn0Kc2F2ZVJEUyhzZXVyYXQub2JqZWN0LCBmaWxlID0gIm1zQWdncl9zY3RkaW0zMy5yZHMiKQpgYGAKCiMjIEV2YWx1YXRlIGNsdXN0ZXIgc3RhYmlsaXR5Ck11c3QgZW5zdXJlIHdlIGhhdmUgdGhlIHJpZ2h0IGNsdXN0ZXIgc3RhYmlsaXR5LCB0aGF0IGlzLCBjZWxscyB0aGF0IHN0YXJ0IGluIHRoZSBzYW1lIGNsdXN0ZXIgdGVuZCB0byBzdGF5IGluIHRoZSBzYW1lIGNsdXN0ZXIuIElmIHlvdXIgZGF0YSBpcyBvdmVyLWNsdXN0ZXJlZCwgY2VsbHMgd2lsbCBib3VuY2UgYmV0d2VlbiBncm91cHMuCgpGb2xsb3dpbmcgW3RoaXMgdHV0b3JpYWwgYnkgTWF0dCBPLl0uaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tLzEwLXRpcHMtZm9yLWNob29zaW5nLXRoZS1vcHRpbWFsLW51bWJlci1vZi1jbHVzdGVycy0yNzdlOTNkNzJkOTIuCgojIyMgQ2x1c3RyZWUKUHJldmlvdXNseSBteSBmYXZvdXJpdGUgaGFzIGJlZW4gQ2x1c3RyZWUsIHdoaWNoIGdpdmVzIGEgbmljZSB2aXN1YWwKTkI6IEZvciBzb21lIHJlYXNvbiBgY2x1c3RyZWU6OmNsdXN0cmVlKClgIGRpZG4ndCB3b3JrLCB3aGVyZWFzIGBsaWJyYXJ5KGNsdXN0cmVlKWAgZm9sbG93ZWQgYnkgYGNsdXN0cmVlKClgIGRpZC4KCmBgYHtyIGZpZy5oZWlnaHQ9NX0KY2x1c3RyZWUoc2V1cmF0Lm9iamVjdCwgcHJlZml4ID0gIlNDVF9zbm5fcmVzLiIsIG5vZGVfY29sb3VyID0gInNjM19zdGFiaWxpdHkiKSArIAoJc2NhbGVfY29sb3JfY29udGludW91cyhsb3cgPSAncmVkMycsIGhpZ2ggPSAnd2hpdGUnKQpgYGAKCkJhc2ljYWxseSBubyBzdGFiaWxpdHkgZGlmZmVyZW5jZSBjb21wYXJlZCB0byBiZWZvcmUuCgoKCgoKIyBEaW0yMgoKQ292ZXIgOTAlIHZhcmlhYmlsaXR5CgpgYGB7cn0KbmRpbSA8LSAyMgpgYGAKCgpgYGB7cn0Kc2V1cmF0Lm9iamVjdCA8LSBSdW5VTUFQKHNldXJhdC5vYmplY3QsIGRpbXMgPSAxOm5kaW0pCnNldXJhdC5vYmplY3QgPC0gRmluZE5laWdoYm9ycyhzZXVyYXQub2JqZWN0LCBkaW1zID0gMTpuZGltKQpmb3IoeCBpbiBjKDAuNSwgMSwgMS41LCAyLCAyLjUpKXsKCXNldXJhdC5vYmplY3QgPC0gRmluZENsdXN0ZXJzKHNldXJhdC5vYmplY3QsIHJlc29sdXRpb24gPSB4KQp9CmBgYApQbG90dGluZy4uLgoKCmBgYHtyfQpmb3IgKG1ldGEuY29sIGluIGNvbG5hbWVzKHNldXJhdC5vYmplY3RAbWV0YS5kYXRhKSl7CglpZihncmVwbChwYXR0ZXJuID0gKCJTQ1Rfc25uX3JlcyIpLCB4ID0gbWV0YS5jb2wpPT1UUlVFKXsKCQlteXBsb3QgPC0gRGltUGxvdChzZXVyYXQub2JqZWN0LCAKCQkJCQkJCQkJCQlncm91cC5ieSA9IG1ldGEuY29sLAoJCQkJCQkJCQkJCXJlZHVjdGlvbiA9ICJ1bWFwIiwgCgkJCQkJCQkJCQkJY29scyA9IGNvbG9yUmFtcHM6OnByaW1hcnkuY29sb3JzKG4gPSBsZW5ndGgobGV2ZWxzKHNldXJhdC5vYmplY3RAbWV0YS5kYXRhW1ttZXRhLmNvbF1dKSkpCgkJCQkJCQkJCQkJKSArIAoJCQlnZ3RpdGxlKHBhc3RlMCgibXNBZ2dyIHNjdCBkaW0iLCBuZGltLCAicmVzIiwgZ3N1YigiU0NUX3Nubl9yZXMiLCAiIiwgbWV0YS5jb2wpICkpCgkJcGxvdChteXBsb3QpCgl9Cn0KYGBgCgoKCgpgYGB7cn0Kc2F2ZVJEUyhzZXVyYXQub2JqZWN0LCBmaWxlID0gcGFzdGUwKCJtc0FnZ3Jfc2N0ZGltIiwgbmRpbSwgIi5yZHMiKSkKYGBgCgojIyBFdmFsdWF0ZSBjbHVzdGVyIHN0YWJpbGl0eQpNdXN0IGVuc3VyZSB3ZSBoYXZlIHRoZSByaWdodCBjbHVzdGVyIHN0YWJpbGl0eSwgdGhhdCBpcywgY2VsbHMgdGhhdCBzdGFydCBpbiB0aGUgc2FtZSBjbHVzdGVyIHRlbmQgdG8gc3RheSBpbiB0aGUgc2FtZSBjbHVzdGVyLiBJZiB5b3VyIGRhdGEgaXMgb3Zlci1jbHVzdGVyZWQsIGNlbGxzIHdpbGwgYm91bmNlIGJldHdlZW4gZ3JvdXBzLgoKRm9sbG93aW5nIFt0aGlzIHR1dG9yaWFsIGJ5IE1hdHQgTy5dLmh0dHBzOi8vdG93YXJkc2RhdGFzY2llbmNlLmNvbS8xMC10aXBzLWZvci1jaG9vc2luZy10aGUtb3B0aW1hbC1udW1iZXItb2YtY2x1c3RlcnMtMjc3ZTkzZDcyZDkyLgoKIyMjIENsdXN0cmVlClByZXZpb3VzbHkgbXkgZmF2b3VyaXRlIGhhcyBiZWVuIENsdXN0cmVlLCB3aGljaCBnaXZlcyBhIG5pY2UgdmlzdWFsCk5COiBGb3Igc29tZSByZWFzb24gYGNsdXN0cmVlOjpjbHVzdHJlZSgpYCBkaWRuJ3Qgd29yaywgd2hlcmVhcyBgbGlicmFyeShjbHVzdHJlZSlgIGZvbGxvd2VkIGJ5IGBjbHVzdHJlZSgpYCBkaWQuCgpgYGB7ciBmaWcuaGVpZ2h0PTV9CmNsdXN0cmVlKHNldXJhdC5vYmplY3QsIHByZWZpeCA9ICJTQ1Rfc25uX3Jlcy4iLCBub2RlX2NvbG91ciA9ICJzYzNfc3RhYmlsaXR5IikgKyAKCXNjYWxlX2NvbG9yX2NvbnRpbnVvdXMobG93ID0gJ3JlZDMnLCBoaWdoID0gJ3doaXRlJykKYGBgCgpIYXJkIHRvIHRlbGwgaWYgZGltMzMgb3IgZGltMjIgaXMgYmV0dGVyLCBidXQgaWYgKmZlZWxzKiBsaWtlIGRpbTIyIGhhcyBsZXNzIGNyb3NzLW92ZXIgYmV0d2VlbiByZXMwLjUgYW5kIHJlczEuMC4gV2lsbCBjb250aW51ZSBhdCBkaW0yMiwgYW5kIHNlZSBob3cgdGhpbmdzIGNoYW5nZSBiZXR3ZWVuIHJlczAuNSBhbmQgcmVzMS4wIAoKYGBge3J9CiMgTnVtYmVyIG9mIGZpbHRlcmVkIGNlbGxzIGxlZnQgaW4gZWFjaCBwb3AKc2FwcGx5KGMoIkxTS20yIiwgIkNNUG0yIiwgIk1FUG0iLCAiR01QbSIpLCBmdW5jdGlvbih4KSAoYyhucm93KHNldXJhdC5vYmplY3RAbWV0YS5kYXRhW3NldXJhdC5vYmplY3RAbWV0YS5kYXRhJG9yaWcuaWRlbnQgPT0geCxdKSkpKQpgYGAKIyMgTWl0b0MgaW4gCgpgYGB7cn0KZm9yICh4IGluIGMoIkxTS20yIiwgIkNNUG0yIiwgIk1FUG0iLCAiR01QbSIpKXsKCWggPSBoaXN0KHNldXJhdC5vYmplY3RAbWV0YS5kYXRhW3NldXJhdC5vYmplY3RAbWV0YS5kYXRhJG9yaWcuaWRlbnQgPT0geCwgJ3BlcmNlbnQubXQnXSwgYnJlYWtzID0gMzAsIHBsb3QgPSBGQUxTRSkKCWgkZGVuc2l0eSA9IGgkY291bnRzL3N1bShoJGNvdW50cykqMTAwCglwbG90KGgsZnJlcT1GQUxTRSwgbWFpbiA9ICBwYXN0ZSh4LCAicGVyY2VudCBtaXRvQyIpLCB4bGFiID0gInBlcmNlbnQgbWl0b0MiLCB5bGFiID0gIkZyZXF1ZW5jeSIpCn0KCgoK